1. Field of the Invention
The present invention relates to an automatic stabilization control apparatus, an automatic stabilization control method, and a recording medium having an automatic stabilization control program recorded thereon.
2. Description of the Background Art
Automatic stabilization control apparatuses for carrying out automatic stabilization control of an image picked up by an image pick-up device such as a video camera have been conventionally known. FIG. 1 is a block diagram showing the automatic stabilization control apparatus. The automatic stabilization control apparatus comprises a frame input unit 50, a geometric conversion coefficient calculation unit 51, an automatic stabilization control unit 52, and a frame output unit 53.
The geometric conversion coefficient calculation unit 51 calculates a geometric conversion coefficient utilizing image processing such as matching from a plurality of frames, for example. The automatic stabilization control unit 52 subjects an input frame which is stored in a memory once to geometric conversion using the geometric conversion coefficient calculated by the geometric conversion coefficient calculation unit 51.
The geometric conversion coefficient is represented by a matrix of 3 by 3, as expressed by the following equation (1) and corresponds to a coefficient in world coordinate conversion between two frames.
                              (                                                                      x                  ′                                                                                                      y                  ′                                                                                    1                                              )                =                              (                                                                                m                    11                                                                                        m                    12                                                                                        m                    13                                                                                                                    m                    21                                                                                        m                    22                                                                                        m                    23                                                                                                0                                                  0                                                  1                                                      )                    ⁢                      (                                                            x                                                                              y                                                                              1                                                      )                                              (        1        )            
FIG. 2 shows the procedure for correction processing in a case where a geometric conversion coefficient is calculated using image processing, and an input frame is subjected to geometric conversion to carry out automatic stabilization control.
Let FS be a start frame (a reference frame), F2 be the current input frame, F1 be an input frame which is one frame preceding the current input frame F2, M12 be a geometric conversion coefficient between the frames F1 and F2 (a geometric conversion coefficient for converting coordinates on the frame F2 into coordinates on the frame F1), MS1 be a geometric conversion coefficient between the frames FS and F1, and MS2 be an accumulated geometric conversion coefficient between the frames FS and F2 (an accumulated geometric conversion coefficient for converting coordinates on the frame F2 into coordinates on the frame FS).
Each of the frames from the start frame to the end frame is subjected to the following processing: That is, an output frame memory is cleared (step 100), the frame is read on a work memory, and the read frame (input frame) is taken as F2 (step 101).
It is then judged whether or not the input frame F2 is the start frame FS (step 102). When it is judged that the input frame F2 is the start frame FS, a coefficient (an initial value) representing non-conversion as expressed by the following equation (2) is set as the accumulated geometric conversion coefficient MS2 between the start frame FS and the input frame F2 (step 103).
                    M        =                  (                                                    1                                            0                                            0                                                                    0                                            1                                            0                                                                    0                                            0                                            1                                              )                                    (        2        )            
The input frame F2 is set as the preceding input frame F1, and MS2 is set as the geometric conversion coefficient MS1 between the start frame FS and the preceding input frame F1 (step 104). Further, data representing the input frame F2 is rendered as it is in the output frame memory (step 105). Thereafter, the procedure is returned to the step 100. In the step 100, the output frame memory is cleared, and the succeeding frame is read.
When it is judged in the foregoing step 102 that the input frame F2 is not the start frame FS, the geometric conversion coefficient M12 between the preceding frame F1 and the current frame F2 is calculated (step 106). The accumulated geometric conversion coefficient MS2 between the start frame FS and the current frame F2 is then calculated from MS1 and M12, as expressed by the following equation (3) (step 107)MS2=M12MS1  (3)
The input frame F2 is set as the preceding input frame F1, and MS2 is set as the geometric conversion coefficient MS1 between the start frame FS and the preceding input frame F1 (step 108). Further, the input frame F2 is subjected to geometric conversion using the accumulated geometric conversion coefficient MS2, and the frame after the geometric conversion is then rendered in the output frame memory (step 109). Thereafter, the procedure is returned to the step 100. In the step 100, the output frame memory is cleared, and the succeeding frame is read.
A typical method of calculating a geometric conversion coefficient between two frames will be described (see JP-A-11-339021). First, a feature point such as an edge is extracted from the one frame. Feature points are then tracked (matched) between the frames, to find a correspondence point on the one frame corresponding to the feature point on the other frame. Two equations such as an equation (4) are obtained from one set of correspondence points (x1, y1) and (x1′, y1′), thereby deriving six or more equations by detecting three or more sets of correspondence points. From the equations, a geometric conversion coefficient with unknowns is calculated using a linear solution.x′=m11x+m12y+m13 y′=m21x+m22y+m23  (4)
In the conventional geometric conversion coefficient calculating method, the accumulated geometric conversion coefficient between the start frame FS and the input frame F2 is calculated by accumulating the geometric conversion coefficients between the adjacent frames from the start frame FS to the input frame F2. Therefore, as the number of input frames increases, calculation errors of the geometric conversion coefficients between the adjacent frames are accumulated. That is, while a large error is made in calculating the geometric conversion coefficient in the frame intermediate between the start frame and the input frame, the frames succeeding the frame corrected by the erroneous conversion coefficient are corrected.
Furthermore, in the conventional automatic stabilization control method, an area where no rendering is performed (hereinafter referred to as a non-rendering area) occurs in the output frame memory. That is, when a plurality of frames picked up in a state where hands move are frames as shown in FIG. 3, frames obtained by rendering data after geometric conversion representing the frames in the output frame memory are as shown in FIG. 4. An oblique portion in each of the frames shown in FIG. 4 is a non-rendering area.